Media processing method and device

ABSTRACT

A media processing system and device with improved power usage characteristics, improved audio functionality and improved media security is provided. Embodiments of the media processing system include an audio processing subsystem that operates independently of the host processor for long periods of time, allowing the host processor to enter a low power state while the audio data is being processed. Other aspects of the media processing system provide for enhanced audio effects such as mixing stored audio samples into real-time telephone audio. Still other aspects of the media processing system provide for improved media security due to the isolation of decrypted audio data from the host processor.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a divisional of U.S. patent application Ser.No. 13/222,871 entitled “Media Processing Method and Device,” filed Aug.31, 2011, which is a divisional of U.S. Pat. No. 8,041,848 entitled“Media Processing Method and Device,” filed Aug. 4, 2008, the entiretyof which is incorporated by reference herein for all purposes.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to electronic devices and, morespecifically, to processing of audio in an electronic device.

2. Description of the Related Art

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present invention,which are described and/or claimed below. This discussion is believed tobe helpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentinvention. Accordingly, it should be understood that these statementsare to be read in this light, and not as admissions of prior art.

The trend in consumer electronics is to combine multiple functionalitiesinto a single portable electronic device. For example, cell phones andmedia players are no longer merely distinct devices, each with their ownunique capabilities. Rather, cell phone and media player functionalitiescan now be merged into one multimedia device with a multitude ofcapabilities. Modern cell phone/media players are often packed withdozens of additional features which include: playing of audio and video,taking of still pictures, recording video, playing video games, GPSnavigation, web surfing, downloading of streaming media from theInternet, Bluetooth and WiFi communications, emailing, text messaging,etc.

One advantage of combining all of these features into one device is thatit eliminates the need to carry multiples devices. From an economicstandpoint, combined devices also reduce overall cost to the consumerbecause the electronics that make up the device are used for multipleapplications rather than having duplicate electronics with specializedfunctions. Additionally, by combining an array of electronics with avariety of capabilities it may be possible to providecross-functionality, in which one device takes advantage of thecapabilities of another device.

Typically, the added functionality of a multimedia device is controlledby a central processing unit (CPU) that has direct access to all of thefeatures provided in the device. For example, in the case of processingstored music, a CPU may directly control the routing of data betweenvarious components such as memory, digital signal processors, decodersand media playing circuitry. In this type of design, most data,including copyright protected media such as music or music videos, willeventually pass through the CPU for processing and routing. The drawbackof this type of design is that the CPU is continually powered up, activeand consuming battery power.

Additionally, the telephone audio in a typical multimedia device may beprocessed by dedicated circuitry rather than the CPU. Generally,telephone audio uses dedicated circuitry to guarantee a hard upper boundon real-time delay, both to comply with the various regulations thatbear upon telephones, and to avoid delays that degrade the userexperience. This may mean that dedicated circuitry is used to processtelephone audio so that telephone audio can bypass the CPU. Thecircuitry dedicated to the processing of telephone audio is typicallyvery simple, limited to equalization and routing functions. The drawbackof this approach is that simplicity of the telephone processingcircuitry limits the type of electronic enhancements of telephone audiothat might otherwise be possible.

Another drawback of combining multiple capabilities in one device isthat as multimedia devices become more functional, the risk ofunauthorized copying and distribution of copyright material becomesgreater. For example, a multimedia device that is capable of downloadingmusic and/or videos from the Internet can also potentially store themedia onto internal memory or an external device and redistribute themedia via email or other Internet communication medium as well as byhard copy. Encryption of copyrighted material may help to make suchmaterial less susceptible to illegal copying; however, in the typicalmultimedia device decrypted media may eventually become available to theCPU and, therefore, vulnerable to illegal copying and distribution.

Thus, typical multimedia or audio devices of the prior art include a CPUthat is directly coupled to all of the audio components, including adigital signal processor (DSP) and peripheral input/output devices. In atypical prior art device, the CPU would be directly involved in many ofthe process steps for processing audio, including routing encoded orcompressed data to a digital signal processor, receiving theuncompressed data from the DSP and routing the uncompressed audio to aperipheral device.

It may be advantageous, therefore, to provide a multimedia device withan audio subsystem that is not directly controlled by the CPU. It wouldalso be advantageous to provide a device that processes a wide varietyof media and takes advantage of the enhanced capabilities that amultimedia device can provide, but, at the same time, provides optimalperformance of a dedicated device. For example, it may be advantageousto provide a multimedia device that combines the capabilities of anaudio player and a cell phone, but also consumes very low power whileoperating as an audio player or a cell phone. Additionally, it may beadvantageous to provide a multimedia device with enhanced copyrightprotection that prevents users from illegally distributing copyrightprotected material.

SUMMARY

Embodiments of the present invention are directed toward a multimediadevice with an independent audio subsystem that is loosely coupled to acentral processing unit (CPU.) In other words, rather than having audiocomponents directly coupled to a main bus, as in the prior art, all ofthe audio components are coupled together separately through anindependent audio bus to form an independent audio subsystem. Thecoupling between the host subsystem and the independent audio subsystemis accomplished through one or more data buffers that allow one waycommunication of data to or from the host subsystem and the audiosubsystem. The audio subsystem is independent in the sense that it doesnot need to further interact with the CPU to accomplish the playing ofaudio data sent to it from the CPU. Rather, when the CPU sends audio tothe audio subsystem, the audio subsystem handles all of the furtherprocessing and routing of the audio. Therefore, the audio subsystemreceives encoded data from the CPU as though the audio subsystem were anoutput device.

Further, embodiments of the present invention are directed toward amultimedia device with an independent audio subsystem that handles allof the decoding, mixing, equalizing and routing of the audio signals,and then sends the output audio directly to peripheral output devices.Because the audio subsystem handles all of the processing of audio data,the CPU is not needed beyond the stage of routing audio data to theaudio subsystem; therefore, the CPU and the rest of the host subsystemmay be configured to enter a low power state while audio data isprocessed. The system may also be configured so that decryption ofprotected media occurs during the transfer of audio data from the hostDMA controller to the audio subsystem. In addition, the audio subsystemmay be configured so that a digital signal processor (DSP) handles therouting, equalizing, and mixing of telephone audio data, and also blendsother audio samples into telephone audio.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentinvention will become better understood when the following detaileddescription of certain exemplary embodiments is read with reference tothe accompanying drawings in which like characters represent like partsthroughout the drawings, wherein:

FIG. 1 is a perspective view of a portable electronic multimedia devicein accordance with an embodiment of the present invention.

FIG. 2 is allow chart showing a general flow of audio data in accordancewith an embodiment of the present invention.

FIG. 3 is a block diagram of a portable electronic multimedia device inaccordance with an embodiment of the present invention.

FIG. 4 is a flow chart of a method for processing music audio inaccordance with an embodiment of the present invention.

FIG. 5 is a flow chart of a method for processing telephone audio inaccordance with an embodiment of the present invention.

FIG. 6 is a flow chart demonstrating a security mechanism in accordancewith an embodiment of the present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

One or more specific embodiments of the present invention will bedescribed below. In an effort to provide a concise description of theseembodiments, not all features of an actual implementation are describedin the specification. It should be appreciated that in the developmentof any such actual implementation, as in any engineering or designproject, numerous implementation-specific decisions must be made toachieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

Turning now to the figures, FIG. 1 depicts an electronic device 100 inaccordance with one embodiment of the present invention. In someembodiments, the electronic device 100 may be a media player for playingmusic and/or video, a cellular phone, a personal data organizer, or anycombination thereof. Thus, the electronic device 100 may be a unifieddevice providing any one of or a combination of the functionality of amedia player, a cellular phone, a personal data organizer, and so forth.In addition, the device 100 may allow a user to connect to andcommunicate through the Internet or through other networks, such aslocal or wide area networks. For example, the electronic device 100 mayallow a user to communicate using e-mail, text messaging, instantmessaging, or using other forms of electronic communication. By way ofexample, the electronic device 100 may be a model of an iPod® having adisplay screen or an iPhone® available from Apple Inc.

In certain embodiments the electronic device 100 may be powered by arechargeable or replaceable battery. Such battery-poweredimplementations may be highly portable, allowing a user to carry theelectronic device 100 while traveling, working, exercising, and soforth. In this manner, a user of the electronic device 100, depending onthe functionalities provided by the electronic device 100, may listen tomusic, play games or video, record video or take pictures, place andtake telephone calls, communicate with others, control other devices(e.g., the device 100 may include remote control and/or Bluetoothfunctionality), and so forth while moving freely with the device 100. Inaddition, in certain embodiments the device 100 may be sized such thatit fits relatively easily into a pocket or hand of the user. In suchembodiments, the device 100 is relatively small and easily handled andutilized by its user and thus may be taken practically anywhere the usertravels. While the present discussion and examples described hereingenerally reference an electronic device 100 which is portable, such asthat depicted in FIG. 1, it should be understood that the techniquesdiscussed herein may be applicable to any media-processing electronicdevice, regardless of the portability of the device.

In the depicted embodiment, the electronic device 100 includes anenclosure 102, a display 104, user input structures 106, andinput/output connectors 108. The enclosure 102 may be formed fromplastic, metal, composite materials, or other suitable materials or anycombination thereof. The enclosure 102 may protect the interiorcomponents of the electronic device 100 from physical damage, and mayalso shield the interior components from electromagnetic interference(EMI).

The display 104 may be a liquid crystal display (LCD) or may be a lightemitting diode (LED) based display, an organic light emitting diode(OLED) based display, or other suitable display. In accordance withcertain embodiments of the present technique, the display 104 maydisplay a user interface 112 as well as various images 105, such aslogos, avatars, photos, album art, and so forth. Additionally, in oneembodiment the display 104 may be a touch screen through which a usermay interact with the user interface. The display 104 may also displayvarious function and/or system indicators to provide feedback to a user,such as power status, call status, memory status, etc. These indicatorsmay be in incorporated into the user interface displayed on the display104. As discussed herein, in certain embodiments the user interface 112may be displayed on the display 104, and may provide a means for a userto interact with the electronic device 100. The user interface may be atextual user interface, a graphical user interface (GUI), or anycombination thereof, and may include various layers, windows, screens,templates, elements or other components that may be displayed in all ofor areas of the display 104.

In one embodiment, one or more of the user input structures 106 areconfigured to control the device 100, such as by controlling a mode ofoperation, an output level, an output type, etc. For instance, the userinput structures 106 may include a button to turn the device 100 on oroff. In general, embodiments of the electronic device 100 may includeany number of user input structures 106, including buttons, switches, acontrol pad, keys, knobs, a scroll wheel, or any other suitable inputstructures. The input structures 106 may work with a user interfacedisplayed on the device 100 to control functions of the device 100 or ofother devices connected to or used by the device 100. For example, theuser input structures 106 may allow a user to navigate a displayed userinterface or to return such a displayed user interface to a default orhome screen.

The user interface 112 may, in certain embodiments, allow a user tointerface with displayed interface elements via the one or more userinput structures 106 and/or via a touch sensitive implementation of thedisplay 104. In such embodiments, the user interface providesinteractive functionality, allowing a user to select, by touch screen orother input structure, from among options displayed on the display 104.Thus the user can operate the device 100 by appropriate interaction withthe user interface 112. The user interface 112 may of any suitabledesign to allow interaction between a user and the device 100. Thus, theuser interface 112 may provide windows, menus, graphics, text, keyboardsor numeric keypads, scrolling devices, or any other elements. In oneembodiment, the user interface 112 may include screens, templates, andUI components, and may include or be divided into any number of these orother elements. The arrangement of the elements of user interface 112may be hierarchical, such that a screen includes one or more templates,a template includes one or UI components. It should be appreciated thatother embodiments may arrange user interface elements in anyhierarchical or non-hierarchical structure.

The electronic device 100 may also include various input and outputports 108 to allow connection of additional devices. For example, a port108 may be a headphone jack that provides for connection of headphones.Additionally, a port 108 may have both input/output capabilities toprovide for connection of a headset (e.g. a headphone and microphonecombination). Embodiments of the present invention may include anynumber of input and/or output ports, including headphone and headsetjacks, universal serial bus (USB) ports, Firewire (IEEE-1394) ports, andAC and/or DC power connectors. Further, the device 100 may use the inputand output ports to connect to and send or receive data with any otherdevice, such as other portable electronic devices, personal computers,printers, etc. For example, in one embodiment the electronic device 100may connect to a personal computer via a Firewire (IEEE-1394) connectionto send and receive data files, such as media files.

The electronic device 100 may also include various audio input andoutput portions. For example, an input receiver 110 may be a microphonethat receives user audio input. Additionally, output transmitter 111 maybe a speaker that transmits audio signals to a user. Input receiver 110and output transmitter 111 may be used in conjunction as audio elementsof a telephone.

Turning to FIGS. 2 and 3, a diagrammatical representation of data flowin the electronic device 100 in accordance with an embodiment of thepresent invention is depicted. As shown in FIG. 3, a typical device 100may include a host subsystem 300 and an audio subsystem 301. In oneembodiment, host subsystem 300 and audio subsystem 301 may be parts of asingle integrated circuit. In another embodiment, host subsystem 300and/or audio subsystem 301 may be distributed over one or moreintegrated circuits. As will be discussed further below, the hostsubsystem 300 includes a CPU 304, which controls and directs mostfunctions of the device 100 other than audio processing. The audiosubsystem 301, on the other hand, controls substantially all of theaudio processing functions of the device 100.

As shown in FIG. 2 and discussed in further detail below, some examplesof audio signals that may be processed by the audio subsystem 301include telephone audio 201, music and audio related to an audio/videosignal 202, and user interface sounds 204. Some audio data, such astelephone audio 201 and user interface sounds 204, may be processeddirectly by the audio subsystem 301 with little or no involvement of theCPU 304, as shown in block 210. Other audio signals, however, may beprocessed by the CPU 304 before being sent to the audio subsystem 301.For example, in block 206, the CPU 304 causes music audio data to bedecrypted and loaded into main memory. Next, at step 208, the CPU 304enters a low power state while audio data is sent to the audio subsystem301. Finally, at step 212, audio data is processed by the audiosubsystem 301 and played, i.e., sent to the appropriate output device.

Because the audio processing at step 212 can take place with little orno involvement of the CPU, the CPU is free to carry out other processingtasks or alternatively to enter a low power state which extends theuseful battery life of the electronic device 100. It should also beappreciated that in other embodiments not depicted by FIG. 2, audio datamay be processed by the audio subsystem then sent to the CPU viavolatile memory 338. For example, the electronic device 100 may includea digital microphone coupled to the audio subsystem, such that audioreceived through the digital microphone may be processed by the audiosubsystem and then sent to the CPU to be processed and/or stored.

Turning now to FIG. 3, a block diagram of a circuitry used in theelectronic device 100 is provided. As seen in the block diagram, theelectronic device 100 includes a main bus 314 to which most of theperipheral electronic components are communicatively coupled. The mainbus 314 may combine the functionality of a direct memory access (DMA)bus and a programmed input/output (PIO) bus. In other words, the mainbus 314 may facilitate both DMA transfers and direct CPU read and writeinstructions. In embodiments of the present invention, the main bus 314may be one of the Advanced Microcontroller Bus Architecture (AMBA®)compliant data buses.

The electronic device 100 also includes a CPU 304. The CPU 304 may beany general purpose microprocessor known in the art such as a ReducedInstruction Set Computer (RISC) from ARM Limited. The CPU 304 runs theoperating system of the electronic device 100 and manages the variousfunctions of the electronic device 100. As such, it may be coupled tothe main bus 314 and configured to transmit PIO instructions to thevarious devices coupled to the main bus 314. Additionally, the CPU 304may be configured to initiate DMA transfers. In one embodiment, PIOinstructions from CPU 304 do not pass directly to the main bus. Rather,as will be explained further below, the CPU 304 is directly coupled tothe main DMA controller 302, and PIO instructions issuing from the CPU304 pass through the main DMA controller 302 to the main bus 314. TheCPU 304 may contain one or more caches whose operation may be completelyinternal to CPU 304, or which may observe and, in some cases, intercept,data transfers passing over main bus 314 that access volatile memory338. The CPU 304 may include or be coupled to a read-only memory (ROM)(not shown), which may hold the operating system and/or other devicefirmware that runs on the electronic device 100.

The electronic device 100 may also include a volatile memory 338electrically coupled to main bus 314. The volatile memory 338 mayinclude, for example, any type of random access memory (RAM), and mayalso include non-volatile memory devices, such as ROM, EPROM and EEPROMor some combination of volatile and non-volatile memory. Additionally,the volatile memory 338 may also include a memory controller thatcontrols the flow of data to and from the volatile memory 338. Inembodiments of the present invention, the volatile memory 338 is used tostore compressed video and/or audio files in encrypted form.

Embodiments of the present invention may also include a main DMAcontroller 302 coupled to the CPU 304 and main bus 314. The main DMAcontroller 302 may be configured to route data between various devicesconnected to the main bus 314 including the volatile memory 338. Therouting of data within the electronic device 100 may be configured tooccur in a variety of ways. For example, the CPU 304 may direct therouting of data through the main DMA controller 302 by creating DMAtransfer instructions in the volatile memory 338, commanding DMAcontroller 302 to begin executing those DMA transfer instructions, andthen commanding an I/O device, attached to main bus 314, to sendtransfer requests, and to transmit and/or receive data from the main DMAcontroller 302. In alternative embodiments, the CPU 304 may route datadirectly by passing data through data registers contained in the CPU304. In other embodiments the electronic device 100 may be implementedwithout a DMA controller, in which case the CPU 304 may directly controlthe flow of data through the electronic device 100 through PIOinstructions.

In addition to the volatile memory 338, the electronic device 100 mayalso include a storage memory 336 connected to the main bus 314. Thestorage memory 336 may include flash memory, such as, for example, NORor NAND flash memory, but may also include any kind of electronicstorage device, such as, for example, magnetic or optical disks. Inembodiments of the present invention, the storage memory 336 is used tostore software and user files such as phonebook entries, pictures, audioand video files, ring tones, archived text messages and emails, etc.

Also coupled to the main bus 314 is an Internet communications device316. The Internet communications device 316 may include any method forcommunicating with the Internet. For example, the Internetcommunications device 316 may include a wireless communications deviceoperating in accordance with IEEE 802.11 standards or an Ethernetcommunication device operating in accordance with IEEE 802.3 standards.In some embodiments, Internet communication device 316 may perform onlya portion of the task of communication with the Internet; for example,in some embodiments Internet communication device 316 may only be thephysical communications link, and the rest of the task of communicationwith the Internet is performed by software executing on CPU 304.

Also connected to the main bus 314 is a user interface 318. The userinterface 318 may include a variety of user interface tools such as, forexample, buttons, knobs, touch screens, trackballs or any other userinterface known in the art.

Also connected to the main bus 314 are video components, including thevideo processing circuitry 308, the video display circuitry 310 and thedisplay 312. The video processing circuitry 308 may be configured tocompress video data into various formats and send the compressed videodata to other parts of the system. For example, the video processingcircuitry 308 may be configured to compress video data obtained fromcamera 306 into a JPEG or MPEG format and send the compressed video datato volatile memory 338 via main bus 314. The video processing circuitry308 may also be configured to decompress video data of various encodingformats and send the decompressed video data to other parts of thesystem. For example, the video processing circuitry 308 may beconfigured to decompress JPEG or MPEG encoded video data obtained fromvolatile memory 338 and send the decompressed video data to the volatilememory 338 or the video display circuitry 310.

The video display circuitry 310 may be configured to convert thedecompressed video data into a video signal that may then be sent to thedisplay 312. The video display circuitry may also be configured togenerate video data in a wide range of video formats. For example, thevideo display circuitry 310 may generate an analog signal such as anNTSC compatible signal or a digital signal such as an ATSC or HDMIcompatible signal. Furthermore, the display 312 may be any type of videodisplay device, such as, for example, an LCD screen. In embodiments ofthe present invention the display 312 is an integral part of theelectronic device 100; however, in alternate embodiments, the display312 may be an external device coupled to the electronic device 100through a data transfer medium such as an HDMI interface, for example.

Together, the video components 308, 310, and 312 may be used to displayvarious forms of video content. For example, the video components 308,310, and 312 may be used to display the real-time camera view throughthe camera 306, or still pictures that have been previously recorded andstored. Additionally, the video components 308, 310, and 312 may be usedto display the video portion of a media with both audio and videocontent. For example, the video components 308, 310, and 312 may be usedto process and display audio/video media such as electronic games orbroadcast media delivered to the electronic device 100 from any possiblesource, such as, for example, a broadcast television signal, streamingmedia from the Internet, or an audio/video file stored in the storagememory 336.

Also connected to the main bus 314 is the data side of the basebandradio 322. The baseband radio 322 transmits and receives wirelesstelephone signals. The data side of the baseband radio 322 is connectedto the host subsystem 300 so that the CPU 304 can directly controlvarious features of the broadband radio 322, such as initiation andtermination of incoming or outgoing phone calls, and so that CPU 304 cantransmit and receive data over a wireless telephone data service (whendoing this, baseband radio 322 has the same role as internetcommunications device 316). As will be explained in further detailbelow, the audio side of the baseband radio is connected to the audiobus 328 so that telephone audio can be processed by the audio subsystemindependently of the CPU 304. In other words, none of the audio datafrom the baseband radio passes to the main bus 314.

FIG. 3 also depicts an embodiment of an independent audio subsystem 301loosely coupled to the host subsystem 300 in accordance of the presentinvention. The audio subsystem is described as “loosely coupled”because, unlike prior art, the components of the audio subsystem are notdirectly coupled to the main bus 314 and are neither directly accessibleby the CPU 304 nor able to directly access main bus 314. Rather, all ofthe audio components included in the audio subsystem 301 are coupled toeach other through an audio bus 328 that is independent from the mainbus 314, and the coupling between the host subsystem 300 and the audiosubsystem 301 is accomplished through a set of data lines controlled byan subsystem interface 320, which will be described further below. Inembodiments of the present invention, the audio bus 328 may be one ofthe AMBA® compliant data buses. Additionally, the audio subsystem 301may also include an audio DMA controller 332 that facilitates DMAtransfers within the audio subsystem 301.

Also coupled to the audio bus 328 is the audio side of the basebandradio 322, which includes a transmitter, a receiver and otherelectronics associated with wireless telephone communications such as,for example, cellular telephone communications. Audio data generated bythe baseband radio 322 is processed by the audio subsystem 301 in amanner that will be described below. Embodiments of the presentinvention may also include other wireless communications devices suchas, for example, a Bluetooth compliant wireless device (not depicted).

Also coupled the audio bus 328 is the CODEC 334, which is configured toencode, decode and route data to and from one or more audio input andoutput devices, such as microphone 110 and speaker 111. Specifically,the CODEC 334 receives analog data from an audio input device such asthe microphone 110, converts the analog audio data into digital audiodata and sends the digital audio data to an audio DSP 330 through theaudio bus 328. Further, the CODEC 334 receives digital audio data fromthe audio DSP 330, converts the digital audio data into analog audiodata and sends the analog audio data to an audio output device such asthe speaker 111. In embodiments of the present invention, the CODEC 334includes two communication channels so that the sending and receiving ofaudio data, as described above, can occur simultaneously.

The audio DSP 330 includes a core processor 333, such as an ARM®AudioDE™ processor, as well as data memory, program memory, DMAchannels, one or more input buffers 329, and one or more output buffers331. In one embodiment, the audio DSP 330 may be an ARM® r2p0 AMCSSprocessor. The audio DSP 330 controls the routing of data within theaudio subsystem 301 and performs various processing of audio data, suchas compression/decompression, equalization and mixing of audio fromdifferent sources. In embodiments of the present invention, the audioDSP 330 is configured to switch between various audio processing tasks,with a grain of a few milliseconds, in order to avoid delays that may beunacceptable by regulations and/or undesirable to users of theelectronic device 100. For example, the audio DSP 330 may be configuredso that if the audio DSP 330 is processing a music audio when the CPU304 initiates an incoming telephone call, the audio DSP 330 will quicklyswitch to the processing of real-time telephone audio in order to avoida delay in the telephone conversation.

The quick switching ability of the audio DSP 330 may be facilitated bythe use of a scheduling hardware and scheduling software running in theprogram memory of the audio DSP 330. In one embodiment, the schedulerbreaks each processing task (e.g., decompression of music audio, orequalization of telephone audio) into a series of smaller task segments,each of which can be processed very quickly. The scheduler thendetermines which task segment is fed to the core processor 333 at anygiven time. Because the scheduler feeds the core processor 333 withsmall task segments that are quickly processed, the scheduler does notneed to interrupt the core processor 333 to switch the core processor333 to a different task. Rather, the scheduler waits until the previoussmall task segment is finished processing before feeding a task segmentrelated to the different task.

Because the most common task performed by the audio DSP 330 will be thedecoding and post-processing (for example, equalization) of audio, thetypical task segment will be decoding or post-processing of a smallsegment of audio samples. The number of samples in the small segment maybe determined by the audio sampling rate for the particular task, aswell as the maximum time delay that can be allowed for the mostdelay-sensitive task, which will usually be the processing of telephoneaudio, as delays in real-time conversation are both forbidden bytelephone regulations and bothersome to the user. Regarding samplingrate, the typical sampling rates will be approximately 8 kilohertz fortelephone audio and 44.1 kilohertz for music. Regarding the maximumallowed time delay, in embodiments of the present invention, the audiosubsystem 301 is configured to process real-time audio, such astelephone audio, with a best-case total time delay of 2 milliseconds(ms) from the time the audio signal is received from the microphone orradio receiver to the time the audio signal is played by the speaker ortransmitted by the radio transmitter. In other embodiments, up to a 5 msdelay may be acceptable. For example, in some embodiments of the presentinvention, the total processing time for real-time audio includes a 1 msdelay due to the input device or receiver, a 1 ms delay due to the audioDSP 330, and a 1 ms delay due to the output device or transmitter. Giventhe above design constraints, the size of the small task segments willtypically be around 8 samples when processing telephone audio, andaround 44-45 samples when processing music.

Also connected to both the main bus 314 and the audio bus 328 is thesubsystem interface 320, which is the main flow path for informationflowing between the host subsystem 300 and the audio subsystem 301,including audio data, control commands, status information andinitialization information. The subsystem interface 320 may include oneor more memory buffers, such as, for example, first-in-first-out (FIFO)buffers or ring buffers, for carrying streaming audio data from the hostsubsystem 300 to the audio subsystem 301. Furthermore, although thesubsystem interface 320 may include a single output buffer channel thatcarries data from the host subsystem 300 to the audio subsystem 301, thesubsystem interface 320 may also include a plurality of buffer channels,including at least one input buffer channel that carries data from theaudio subsystem 301 to the host subsystem 300. In one embodiment, thesubsystem interface 320 includes four buffer channels that can beindividually configured as input or output and are usually configured asthree output buffer channels and one input buffer channel. By providingmore than one buffer channel to carry data to the audio subsystem 301,streaming audio, such as music audio, can be kept separate from userinterface sounds. The subsystem interface 320 may also include one ormore electronic registers, used to carry control information from theCPU 304 to the audio subsystem 301 and to carry status information fromthe audio subsystem 301 to the CPU 304. Both the CPU 304 and the audioDSP 330 may have read/write access to these registers.

Additionally, the audio subsystem 301 may also include an amount ofaudio RAM, or other form of electronic data storage, sufficient totemporarily store a significant quantity of streaming audio data ineither compressed or un-compressed format that has been sent by hostsubsystem 300 and is waiting to be processed by the audio DSP 330. Insome embodiments, the audio RAM may be included in the subsysteminterface 320. In other embodiments, the audio RAM may be included inthe audio DSP 330, in which case the input buffer 329 and the outputbuffer 331 may be included in the audio RAM. In yet other embodiments,the audio RAM may be a separate component coupled to the audio bus 328.By providing an audio RAM to temporarily store streaming audio, the hostsubsystem 300 can go into a low power mode while the audio DSP continuesto process audio data, as will be explained in more detail below. Insome embodiments, the audio subsystem 301 may include a relatively smallamount of audio RAM, on the order of ten kilobytes or less. In otherembodiments, the audio subsystem 301 may include several hundredkilobytes of audio RAM. It will be appreciated that increasing theamount of audio RAM included in the audio subsystem 301 will increasethe length of time the host subsystem 300 can remain in low power mode.In some embodiments, the audio RAM may hold the equivalent of ten tothirty seconds worth of audio data.

Additionally, a means of synchronizing audio data with video data isprovided. Synchronization of audio and video data may be desirablebecause, in embodiments of the present invention, the audio and videocomponents of an audio/video signal are processed separately.Specifically, the video component of an audio/video signal is sent tothe video processing circuitry 308, while the audio component of theaudio/video signal is sent to the audio subsystem 301. Furthermore, onceaudio data is sent to the audio subsystem 301, the CPU 304 is unable toretrieve or manipulate the audio data. Rather, audio data processed bythe audio DSP 330 may be played by sending the audio signal to the CODEC334 or some other audio output device within the audio subsystem 301such as an S/PDIF output or the audio/video interface 326. Therefore,without a means to synchronize the playing of audio and video data, theunequal processing times that may exist between the audio processingcircuitry and the video processing circuitry could cause the videocomponent to play out of synch with the audio component.

To make it possible for software running on the host subsystem 300 andthe audio processing subsystem 301 to ensure that the audio and videocomponents of an audio/video signal play with the correct timing, atimekeeping device 324 is connected to both the main bus 314 and theaudio bus 328. The timekeeping device 324 may include a master timebase,a register that increments continuously at a constant frequency. Boththe host subsystem 300 and the audio subsystem 301 have access to thetime information generated by the timekeeping device 324. This timinginformation may then be used to synchronize audio output with videooutput when the electronic device 100 is generating audio/video output.For example, the audio component of an audio/video may be encoded with atimestamp before being sent to the audio subsystem 301. The timestampwould inform the audio subsystem 301 of the appropriate time to play thesegment of audio, with the master timebase serving as the authoritativesource of the current time. For another example, timing information maybe read from the timekeeping device 324 and written to a data registerin the subsystem interface 320 when audio samples corresponding tointeresting points in an audio stream are delivered to CODEC 334. Thetiming information may then inform the CPU 304 of the time that aparticular audio sample has been played by the CODEC 334.

The audio/video interface 326 provides a means for recombining theprocessed audio and video in embodiments in which the audio/video signalis sent to a display device that can utilize a combined audio/videosignal. For example, the audio/video interface 326 may convert the audiodata into a High-Definition Multimedia Interface (HDMI) format. Audiodata sent through the audio/video interface 326 to the video displaycircuitry 310 may include timing information read from the timekeepinginterface 324. Because this timing information is common to both thehost subsystem 300 and the audio sub-system, the information can be usedto synchronize the audio and video signals.

It should be noted that various configurations and capabilities of theelectronic device 100 may be utilized in accordance with embodiments ofthe present invention. As an example, embodiments of the presentinvention may be configured without the video processing circuitry 308,the video display circuitry 310, or the display 312. Additionally,embodiments of the present invention may support a variety of inputmedia such as Ethernet or USB. Regarding the audio subsystem 301, theelectronic device 100 may include, among other things, a digitalmicrophone connected to the audio bus 328, configured to allow a user torecord audio samples or voice commands. Although it is beyond the scopeof the present description to detail every possible combination ofcomponents that may be included in the electronic device 100 and theaudio subsystem 301, it will be appreciated by those of ordinary skillin the art that various other components may be added or eliminatedwithout deviating from the spirit and scope of the present invention. Itshould also be noted that some or all of the components described inFIG. 3 may be implemented in a system on a chip (SOC). Furthermore, insome embodiments, the audio subsystem may be implemented in its ownseparate chip.

Additionally, it will also be appreciated that certain embodiments mayalso include the processing of other types of media, such as video,within a loosely coupled subsystem. For example, the video processingcircuitry 308, camera 306, video display circuitry 310 and display 312may be coupled together by a video bus to form a second subsystem whichcommunicates with the host subsystem 300 through a second subsysteminterface located between the main bus 314 and the video bus. Foranother example, the video and audio components may all be included inthe audio subsystem 301, in which case the audio subsystem 301 mayactually be an audio/video subsystem. For convenience, the presentapplication describes an electronic device with a loosely coupled audiosubsystem 301. It will be understood, however, that embodiments of thepresent invention also apply to a device with a loosely coupled videosubsystem.

Turning now to FIG. 4, a method for processing music audio in accordancewith an embodiment of the present invention is depicted. The describedembodiment includes two parallel processes, process 400 and process 401,which are substantially independent from one another. Process 400includes the processing steps carried out by the host subsystem 300,while process 401 includes the processing steps carried out by the audiosubsystem 301.

Process 400 begins at step 402, in which the user initiates the playingof an audio track. The playing of the audio track may be initiated bythe selection of a particular music or audio/video file or may beinitiated by the selection of a particular Internet web page, forexample.

At step 404, the CPU 304 reads a large block of audio data from storagedevice (e.g., 336) and writes the data into memory, such as the volatilememory 338. The large block of audio data may include several minutes ofaudio data made up of a single audio track, several audio tracks or evenpart of an audio track. Additionally, the audio data may originate froma wide variety of sources. For example, audio data may originate from astorage device such as the storage memory 336, or may stream from theInternet via the Internet communications device 316. After reading thelarge block of data (e.g., from non-volatile storage 336) and writingthe data into memory, the CPU 304 creates DMA transfer instructions involatile memory 338 which provides DMA controller 302 with a descriptionof the block of audio data, sends the DMA transfer instructions to DMAcontroller 302, and initiates a DMA transfer.

Next, at step 406, the CPU 304 enters into a low power state. Those ofordinary skill in the art will recognize several known methods forachieving a low power state. For example, the CPU 304 may switch offsome or all of its internal circuitry by electrically decoupling thecircuitry from the main power source. Alternatively, some or all of theCPU 304 may stop its clocks using a clock gating technique, which willbe known to those of ordinary skill in the art. During step 406, the CPU304 may also power down various other electronic components within theelectronic device 100 that are not used for playing the selected audio.In some embodiments, at least one data channel within the CPU 304, suchas an interrupt channel, remains active so that the CPU 304 can betriggered to resume normal operation through the data channel. In thisway, the CPU 304 may be triggered to resume normal operation upondetection of a certain event, such as the initiation of an incoming oroutgoing phone call, for example.

Next, at step 408, while the CPU 304 remains in a low power state, themain DMA controller 302 transfers audio data from the volatile memory338 to the audio subsystem 301 via the subsystem interface 320 inaccordance with the DMA transfer instructions. The main DMA controller302 may be configured to transfer as much audio data as the RAM in thesubsystem interface 320 can hold, which may be up to several secondsworth.

Next, at step 410, the main DMA controller 302 sends a signal to theAudio DSP 330 indicating that data is available in the subsysteminterface 320. The sending of this signal, or handshake, triggers theparallel process 401, wherein the audio DSP 330 processes the audio dataindependently of the host subsystem 300, which will be described furtherbelow.

Next, process 400 advances to step 412, wherein both the main DMAcontroller 302 and the volatile memory 338 enter a lower power state. Aswith the CPU 304, the low power state of the main DMA controller 302 maybe implemented by electrically decoupling the main DMA controller 302from its power supply, through clock gating, or by any otherpower-saving technique known by those of ordinary skill in the art. Ifany portion of the audio to be played is still in the volatile memory338, waiting to be transferred, the low power state of the volatilememory 338 is implemented by any technique, which preserves the datacontained in volatile memory. For example, if the volatile memory 338 isimplemented using dynamic RAM (DRAM) technology, then volatile memory338 may be placed into a self-refresh mode.

At this time, unless a parallel task is being carried out by the CPU304, the audio subsystem 301 is substantially the only component withinthe electronic device 100 that is fully powered. As will be discussed inrelation to process 401, the audio subsystem 301 may continue to drawaudio data from the subsystem interface 320 independently of the hostsubsystem 300 and process 400. The host subsystem 300 may, therefore,power up and send new audio data to the subsystem interface 320 beforethe subsystem interface 320 runs out of audio data.

Accordingly, at step 414, a determination is made as to whether theaudio data stored in the subsystem interface 320 has fallen below aspecified threshold. If the audio data is not below the specifiedthreshold, the main DMA controller 302 remains in the low power state,as indicated at step 416. If however, the audio data is below thespecified threshold, a process is initiated by which the next portion ofthe selected audio track is loaded into the subsystem interface 320. Insome embodiments, step 414 is executed periodically after a specifiedamount of data has been transferred from the subsystem interface 320 tothe audio DSP 330.

After the audio data stored in the subsystem interface 320 falls belowthe threshold, the process advances to step 418, at which stage thesubsystem interface 320 sends a DMA request to the main DMA controller302, causing the main DMA controller 302 to power up and resume normaloperation. Next, at step 420, a determination is made as to whether themain DMA controller 302 has reached the end of the DMA transferinstructions. If the main DMA controller 302 is not at the end of theDMA transfer instructions, process 400 then advances to step 408, inwhich case the main DMA controller 302 transfers more audio data to thesubsystem interface 320, sends a handshake signal to the audio DSP 330,and goes back into low power state (steps 408, 410 and 412). If the mainDMA controller 302 is at the end of the descriptor, this indicates thatall of the audio data stored in host memory by the CPU 304 at step 404has been transferred to the audio subsystem 301. Therefore, if the mainDMA controller 302 has reached the end of the DMA transfer instructions,process 400 advances to step 422 to begin the process of loading anotherblock of audio data.

At step 422, the main DMA controller 302 sends an interrupt command tothe CPU 304, thereby causing the CPU 304 to power up and resume normaloperation. Next, at step 404, the CPU 304 repeats the process detailedabove. Namely, the CPU 304 writes another large block of audio data intomemory, creates DMA transfer instructions in volatile memory 338,initiates a DMA transfer (step 404) and drops back into a low powerstate (step 406). In embodiments of the present invention, the new blockof audio data may be the remainder of a particular song that wasselected by the user, or alternatively, the new audio data may be from anew song, such as for example, the next song stored in memory, or thenext song in a list of songs specified, in some way, by the user.

Turning now to process 401, the processing of the audio data within theaudio subsystem 301 will be described. Process 401 begins at step 424,wherein the audio DSP 330 waits for audio data to become available atthe subsystem interface. The handshake signal sent by the main DMAcontroller 302 at step 410 triggers process 401 to advance from step 424to step 426.

At step 426, the audio DSP 330 moves the audio data from the subsysteminterface 320 to the audio DSP 330 memory, such as the input buffer 329.In some embodiments, as will be explained further below, the datatransfer may not occur immediately if the audio DSP 330 is busyprocessing a higher priority task, such as processing and routing ofreal-time telephone audio. Otherwise, the audio data transfer will occuras soon as the subsystem interface 320 contains a specified amount ofaudio data.

Next, at step 428 the audio data transferred to the input buffer 329 ofthe audio DSP 330 is processed by the core processor 333 of the audioDSP 330, such as by decoding, equalizing and/or mixing the audio data.For example, the audio DSP 330 will typically decode the audio data toconvert the audio data into a format recognized by the CODEC 334. Thedecoding process is usually necessary because, in some embodiments, theaudio data is transferred to the audio subsystem 301 in a compressedformat and must, therefore, be decompressed before being sent to theCODEC 334. For another example, the audio DSP 330 may also have toequalize the audio data. The equalization process may be necessary tocorrect for signal distortion introduced by CODEC 334, microphone 110,or speaker 111. For yet another example, the audio DSP 330 may mix twoaudio data streams together before sending them to the CODEC 334. Themixing process provides a means of incorporating a wide variety of audiofeatures into the electronic device 100. For example, interface soundsmay be sent by the CPU 304 and mixed with playing music, or audio may befaded smoothly between music tracks. Additionally, because the telephoneaudio and music audio are both handled by the audio subsystem 301,various audio effects, which combine telephone and music audio, may berealized. For example, in one embodiment, music and telephone audio maybe mixed to create a smooth fading between the music and telephone audioupon the initiation of a phone call. In another embodiment, music andtelephone audio may be mixed to create background music that plays at alower volume throughout a telephone conversation.

As stated above, the audio data may not be immediately processed if ahigher priority task is currently using the core processor 333 of theaudio DSP 330. Consequently, if the input buffer 329 of the audio DSP330 becomes full, the subsystem interface 320 will stop sending audiodata to the audio DSP 330, resuming only when the input buffer 329 canhandle new data. As stated above, embodiments of the present inventioninclude scheduling hardware and scheduling software running in theprogram memory of the audio DSP 330 that selects which task getsprocessed at any time, given the assigned priority of the task. When thescheduler selects the audio data for processing step 428 may execute.The playing of audio data may, therefore, pause at step 428 while ahigher priority task is completed. For example, if an incoming call isinitiated while an audio track is playing, process 401 will pause atstep 428, resuming when the phone call has ended.

Next, at step 430, the audio DSP 330 stores the processed audio data inaudio output buffer 331 and routes the processed audio data from theoutput buffer 331 to the CODEC 334. The CODEC 334, as discussed above,then converts the digital audio data into an analog signal and sends theanalog audio signal to an output device such as a speaker or headphone.Alternatively, depending on the type of audio and/or the user'sselection, the audio DSP 330 may send the processed audio to aperipheral audio device other than the CODEC 334. For example, the audioDSP 330 may route data to the audio/video interface 326 or an SPDIFinterface.

In some embodiments of the present invention, steps 424, 426, 428 and430 repeat until all of the audio data transferred to the subsysteminterface 320 is processed and played, i.e. routed to the output device.It should be noted that, in the presently described embodiment, audiodata sent to the audio subsystem 301 does not have access to a returnpath back to the host subsystem 300; therefore, audio data sent to theaudio subsystem is no longer visible to the host subsystem 300,including the CPU 304. Furthermore, process 400 may repeat until all ofthe audio data selected by the user has been processed by the audiosubsystem 301 and played, or until the process has been cancelled by theuser.

Turning now to FIG. 5, a method of processing telephone audio inaccordance with an embodiment of the present invention is depicted.Process 500 starts at step 502 with the initiation of a telephone call.The telephone call may be an incoming call or an outgoing call initiatedby the user. In embodiments of the present invention, step 502 istriggered when the user has committed to making or receiving a telephonecall. by either dialing a call or pressing the “answer” button.

Next, at step 504, the host subsystem 300 enters a low power mode. Thehost subsystem 300 may include the CPU 304, the main DMA controller 302,the storage memory 336, the volatile memory 338, and/or any othercomponent connected to the main bus 326 that is not part of the audiosubsystem 301. In this way, only those components used for theprocessing of telephone audio are fully powered. As stated above, thelow power mode may be implemented using any techniques known on the art,such as clock gating. Additionally, although process 500 depicts thehost subsystem 300 as remaining in low power mode for the duration ofthe telephone call, the host subsystem 300 or some part thereof, suchas, for example, the CPU 304, may resume normal operation if triggeredto do so. For example, if the user selected music audio or a sound clipto play during a telephone conversation, the host subsystem 300 mayresume normal operation to deliver the selected audio to the audiosubsystem 301. Additionally, the host subsystem 300 may not go into lowpower state at all if host subsystem 300 has been configured by the userto carry out a parallel process during the processing of telephoneaudio.

Next, at step 506, it is determined whether the baseband radio 322 andthe CODEC 334 are “ready,” meaning that they are ready to send andreceive audio data. The test for the “ready” condition may be physicallyimplemented by any means known in the art. For example, the CODEC 334may include an output buffer, which holds data to be sent to the outputaudio device, and an input buffer, which holds data that has beenreceived by the audio input device and will be sent to the audio DSP 330for processing. Similarly, the baseband radio 322 may include an outputbuffer, which holds outgoing data to be transmitted, and an inputbuffer, which holds incoming data that has been received by the basebandradio 322 and will be sent to the audio DSP 330 for processing. In anembodiment of the present invention, the “ready” condition may signifythat both output buffers contain a specified number of empty memoryspaces and that both input buffers contain a specified number of audiosamples. If all four of the buffers indicate a “ready” state, then atransfer of data may take place and process 500, therefore, advances tostep 508. Waiting for all four of the buffers to indicate a “ready”state simplifies subsequent processing, since subsequent processingsteps can assume that any input data is available, and space for anyoutput data is available; it is not necessary to check for the presenceof input data, or the availability of output buffer space, as theprocessing proceeds. In other embodiments, the presence of input dataand the availability of output buffer space are separately determined bythe audio DSP 330 during processing.

Step 506 prioritizes the processing of real-time telephone audio, whilealso allowing efficient use of the audio DSP's 330 core processor 333.As long as telephone audio is waiting to be processed and the telephonerelated circuitry is capable of handling new data, the audio DSP 330circuitry will be applied to processing telephone audio. Prioritizingthe real-time telephone audio helps to ensure that telephone audio isprocessed quickly, thereby avoiding delays in telephone audio that maybe forbidden by regulations and/or undesirable to the user. There may beshort periods of time, however, when the telephone circuitry cannot makeuse of the audio DSP 330. This may happen, for example, if the outputbuffers of the baseband radio 322 or the CODEC 334 are full or close tofull, or if the input buffers of the baseband radio 322 or the CODEC 334are empty or close to empty. Rather than allow the audio DSP 330 toremain idle during these times, embodiments of the present inventioninclude a process by which the audio DSP 330 processes other tasks whenthe baseband radio 322 or the CODEC 334 are not in a “ready” state. Assuch, process 500 may branch at step 506, thereby proceeding to step 520if either the baseband radio 322 or the CODEC 334 are not in a “ready”state.

If all four of the buffers indicate a “ready” state, process 500advances to step 508. At step 508, a request is made to the audio DSP330. In one embodiment, the request is a DMA request, and the request isimplemented by DMA request circuitry that includes a set of four datalines each coupled to one of the four buffers and configured to indicatewhether the buffer is in a “ready” state. The four data lines may befurther coupled to the input of an “and” gate, with the output of the“and” gate coupled to a DMA request input line included in the audio DSP330. In another embodiment, the request is an interrupt request, and therequest is implemented by interrupt request circuitry that includes aset of four data lines each coupled to one of the four buffers andconfigured to indicate whether the buffer is in the “ready” state. Thefour data lines may be further coupled to the input of an “and” gate,with the output of the “and” gate coupled to an interrupt request inputline included in the audio DSP 320.

Next, at step 510, telephone audio data is transferred from the inputbuffers of the CODEC 334 and the baseband radio 322 to the input buffers329 of the audio DSP 330. In one embodiment, the audio DSP 330 includesat least two input buffers, and the transfer of data from the CODEC 334and the baseband radio 322 occurs simultaneously. In one embodiment,where the request was a DMA request, the transfer would be performed bya DMA controller in response to the DMA request. In another embodiment,where the request was an interrupt request, the transfer would beperformed by programmed I/O operations generated by software running onaudio DSP 330 in response to the interrupt request.

Next, at step 512, telephone audio is taken from the input buffers 329of audio DSP 330, processed by the audio DSP 330, and sent to an outputbuffer 331 of the audio DSP 330. Examples of such processing may includedecoding or encoding of incoming or outgoing transmissions, equalizationof telephone audio and digital mixing of different audio data streams.Those of ordinary skill in the art will recognize methods of carryingout the above described processes.

Next, at step 514, the audio DSP 330 may mix other audio samples intothe telephone audio. For example, the CPU 304 may cause a user interfacesound such as a beeping or ringing sound to be mixed with telephoneaudio as an indication of another incoming call. For another example,decoded music audio may be mixed with telephone audio, so that decodedmusic audio plays in the background of the incoming and/or outgoingtelephone audio. In order to mix decoded music audio with telephoneaudio, the sample rate of the music audio may be converted by the audioDSP 330 to match the sample rate of the telephone audio, as will beappreciated by those of ordinary skill in the art.

Next, at step 516, telephone audio data is sent to the output buffers ofthe broadband radio 322 and the CODEC 334. In the embodiment presentlydescribed, the transfer of data to the CODEC 334 and the baseband radio322 occurs simultaneously. Therefore, according to one embodiment, theaudio DSP 330 also includes at least two output buffers 331. In oneembodiment, the transfer would be performed by a DMA controller. Inanother embodiment, the transfer would be performed by programmed I/Ooperations generated by software running on audio DSP 330.

Next, at step 518, the baseband radio 322 transmits the outgoing audiodata, and the CODEC 334 sends the incoming audio data to an outputdevice, such as a speaker. Those of ordinary skill in the art willrecognize techniques for carrying out the processes described in step518.

In embodiments of the present invention, steps 506 through 518 repeat ata rate, which is determined by the rate at which the baseband radio andthe codec become “ready,” until all of the incoming and outgoing audiodata are processed and the telephone call ends. Once the telephone callends, the host subsystem 300 comes out of low power mode and resumesnormal operation. As stated above, the host subsystem 300 may resumenormal operation even while a telephone call is ongoing, if triggered todo so. Additionally, the host subsystem 300 may also stay in low powermode after the termination of the telephone call if there is nosubsequent task waiting to be processed by the host subsystem 300.

Returning to step 506, as stated above, if one of the four buffers doesnot indicate a “ready” state, process 500 will advances to step 520. Atstep 520, the audio DSP 320 may perform some other short processingtask, such as decoding a segment of audio data that has been transferredfrom the subsystem interface 320, or processing user interface sounddata sent by the CPU 304 to alert the user of another incoming call or alow battery condition, etc. The processing that takes place at step 520will be brief enough that the next scheduled execution of telephoneaudio processing is not significantly delayed. Because the audio DSPprocesses data in small task segments, as discussed above, step 520 willfinish and process 500 will return to step 506 quickly. The audio dataprocessed during step 520 may eventually be mixed with telephone audioduring step 514.

Turning now to the enhanced security attributes of the presentinvention, embodiments of the present invention include techniques forenhancing the security of copyrighted audio through encryption. Examplesof such encryption techniques are discussed in the copending andcommonly assigned U.S. patent application Ser. No. 12/060,728, filedApr. 1, 2008, entitled “Central DMA with Arbitrary ProcessingFunctions,” which is hereby incorporated by reference in its entirety.Briefly stated, copyrighted audio data loaded into the electronic device100 may be stored in memory in encrypted form, thereby rendering anyunauthorized copies that may be downloaded from the electronic device100 inoperable. The electronic device 100 is, therefore, equipped withdecryption hardware and/or software so that encrypted audio files may bedecrypted before playing. Some decryption processes could, however,present an opportunity for a hacker to download the un-encrypted audiofile from a memory location in the electronic device 100 such as theinternal registers of the CPU 304 or the volatile memory 338. It istherefore beneficial to use a decryption process in which the decryptedaudio data does not appear in a memory location that is accessible toCPU 304. To achieve this, the encryption techniques discussed in U.S.patent application Ser. No. 12/060,728 cause the decryption of audiodata to occur simultaneously with the sending of audio data from memoryto the buffers of the target DMA device.

In embodiments of the present invention, the above-referencedencryption/decryption techniques are employed in the electronic device100, to enhance the security of copyright protected material. Forexample, decryption of encrypted audio data may be performed by the mainDMA controller 302 as part of the process of transferring the audio tothe subsystem interface 320. In this way, decrypted audio data appearson the main bus 314 and within the audio subsystem 301, but not in amemory location to which the CPU 304 has access. The CPU 304 does nothave access to the audio sent to the audio subsystem 301, becauseembodiments of the present invention are configured to only allow oneway transmission of decoded audio data to the audio subsystem 301. Inother words, the audio subsystem 301 will appear to the CPU 304 as anoutput device coupled to the main bus, to which compressed audio data issent to be processed and played. This functionality is achieved, inpart, by the audio DSP 330, which not only decompresses compressedaudio, but also equalizes, mixes and routes audio data to the CODEC 334,thereby eliminating the CPU 304 from processes involving the handling ofdecoded audio data.

Therefore, in embodiments of the present invention the CPU 304 haslimited access to the resources within the audio subsystem 301 includingthe audio DSP 330. This limited access is achieved by the design of thesubsystem interface 320. The subsystem interface 320 is configured suchthat, during operation of the electronic device 100, the CPU 304 hasaccess only to the host side of the subsystem interface 320, which, asdiscussed above, allows CPU 304 to access only a subset of the resourceswithin audio subsystem 301. For example, in one embodiment, thesubsystem interface 320 is configured to allow access to only a portionof the volatile memory included in audio DSP 330, including input buffer329 and one or more control and/or status registers. The bulk of thevolatile memory included in the audio subsystem 301 is, therefore, notaccessible to the CPU 304. Embodiments of the present invention,therefore, make it difficult for a hacker to create unauthorized copiesof copyrighted material, because the CPU 304 does not have access todecrypted audio data.

Additionally, to inhibit the ability of a hacker to load unauthorizedsoftware on the device 10, the host subsystem 300 may periodicallyperform an audit of the software loaded in each of the devices coupledto the main bus 300. Because the audio subsystem 301 does not haveaccess to the main bus 300, none of the code in the audio subsystem 301needs to be considered. This saves in processing time required for theaudit.

Although the host subsystem 300 has limited access to the audiosubsystem 301, it may also be necessary for the host CPU 304 totemporarily have broad access to the audio subsystem 301 when theelectronic device 100 is initially powered up. This temporary access mayallow the CPU 304 to load certain initialization data used by the audiosubsystem 301. For example, the CPU 304 may load the scheduler softwarethat runs on the audio DSP 330 when the electronic device 100 is poweredup. Additionally, the CPU may also provide initialization data to othercomponents on the audio bus when the electronic device 100 is poweredup. In order to load the audio DSP 330 software and other initializationdata, the CPU 304 has temporary broad access, during a briefinitialization stage, to the memory address space contained in the audioDSP 330, the subsystem interface 320, and any other component to beserviced by the CPU 304 on powering up. However, to maintain thesecurity of decrypted audio, the CPU's 304 access to the internalresources of audio subsystem 301 using the subsystem interface 320 issubsequently restricted after the brief initialization stage.

To maintain the security of the audio subsystem 301 while still allowingthe CPU 304 to load software and initialization data, an embodiment ofthe present invention includes a security mechanism built into thesubsystem interface 320. The security mechanism allows the subsysteminterface 320 to operate in one of two modes: “insecure mode” and“secure mode.” In “insecure mode,” the CPU 304 has full access to theinternal resources of the audio subsystem 301 using subsystem interface320. In “secure mode” CPU 304 has restricted access to the internalresources of the audio subsystem 301 using subsystem interface 320. Thesubsystem interface 320 is forced into “insecure mode” when theelectronic device 100 is powered up, and software can explicity switchthe subsystem interface 320 from “insecure mode” to “secure mode”, butthere is no way (other that powering the electronic device 101 down andup) to return to “insecure mode.”

FIG. 6 depicts the operation of a security mechanism built into thesubsystem interface 320, in accordance with an embodiment of the presentinvention. Method 600 starts when the electronic device 100 is poweredup at step 602. Immediately upon powering up, the CPU 304 runs trustedcode, code that is known to be safe and uncorrupted. An example oftrusted code is code that is loaded from read only memory (ROM) locatedon the same integrated circuit that contains CPU 304.

Next, at step 604, the trusted CPU code loads software into the programmemory space of the audio DSP 330 through the subsystem interface 320.During step 604, the CPU 304 may also optionally load any otherinitialization data or software used by other components in the audiosubsystem 301. During step 604, the CPU 304 may have unrestricted accessto substantially all of the memory address space available on the audioDSP 330, the subsystem interface 320, and any other component attachedto the audio bus 328.

Next, at step 606, after all initialization data is loaded, the audiosubsystem software is started. After the audio subsystem software isstarted, the initialization path is disabled at step 608. In oneembodiment, disabling the initalization path means switching subsysteminterface 320 from “insecure mode” to “secure mode.” In one embodiment,the audio DSP 330 itself disables the initialization path according tothe software loaded by the CPU 304. In other embodiments, theinitialization path is disabled by the CPU 304. When the initializationpath is disabled, the CPU 304 no longer has unrestricted access to theinternal resource of the audio subsystem 301. For example, after theinitialization path is disabled, the CPU 304 may only access the portionof the volatile memory included in audio DSP 330 that includes inputbuffer 329 and one or more control and/or status registers. Once theinitialization path is disabled, an attempt by the CPU 304 to access arestricted memory address simply returns an error. Furthermore, once theinitialization path is disabled, it cannot be enabled again, other thanby turning the electronic device 100 off and on again, in which case,process 600 repeats.

While the invention may be susceptible to various modifications andalternative forms, specific embodiments have been shown by way ofexample in the drawings and have been described in detail herein.However, it should be understood that the invention is not intended tobe limited to the particular forms disclosed. Rather, the invention isto cover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention as defined by the followingappended claims.

What is claimed is:
 1. A digital signal processor, comprising: a coreprocessor configured to process audio data; at least one input buffercoupled to the core processor, the at least one input buffer configuredto receive audio data from at least one audio input device; at least oneoutput buffer coupled to the core processor, the at least one outputbuffer configured to send processed audio data to at least one audiooutput device; and a programmable memory configured to interface withthe core processor to control the routing of audio data segments fromthe at least one input buffer to the at least one output buffer, whereincontrolling the routing of the audio data segments comprises determiningan order in which the audio data segments are processed by the coreprocessor and routed to the output buffer.
 2. The digital signalprocessor of claim 1, wherein the programmable memory is configured tointerface with the core processor to assign a priority level to each ofthe audio data segments.
 3. The digital signal processor of claim 1,wherein the programmable memory is configured to interface with the coreprocessor to control the routing of audio data from the at least oneaudio input device to the at least one input buffer.
 4. The digitalsignal processor of claim 1, wherein the programmable memory isconfigured to interface with the core processor to control the routingof processed audio data from the at least one output buffer to the atleast one audio output device.
 5. The digital signal processor of claim1, wherein the core processor is configured to mix two or more audiosamples; sample rate convert audio data; equalize audio data; compressor decompress audio data; encrypt or decrypt audio data; or somecombination thereof.
 6. The digital signal processor of claim 1, whereinthe at least one audio input device comprises one or more microphones, aheadset, or a combination thereof.
 7. The digital signal processor ofclaim 1, wherein the at least one audio output device comprises aspeaker, a headphone, a headset, or a combination thereof.
 8. Thedigital signal processor of claim 1, wherein the at least one inputbuffer is configured to receive the audio data from the at least oneaudio input device comprising first receiving the audio data from aCODEC, wherein the CODEC is configured to convert the audio data intodigital audio data.
 9. The digital signal processor of claim 8, whereinthe CODEC comprises dual communication channels, and wherein the CODECis configured to convert the audio data into digital audio data andconvert the processed audio data into analog audio data substantiallysimultaneously.
 10. The digital signal processor of claim 1, wherein theat least one output buffer is configured to send the processed audiodata to the at least one audio output device comprising first sendingthe processed audio data to a CODEC, wherein the CODEC is configured toconvert the processed audio data into analog audio data.
 11. A method ofprocessing audio data via a digital signal processor (DSP), comprising:receiving audio data into an input buffer; partitioning the audio datainto audio data segments via a programmable memory configured tointerface with a core processor of the DSP, wherein the audio datasegments correspond to a plurality of audio processing task segments;assigning a respective priority level to each of the plurality of audioprocessing task segments; processing each audio processing task segmentof the plurality of audio processing task segments via the coreprocessor according to the respective priority level of each of theplurality of audio processing task segments; and storing the audio datasegments into an output buffer.
 12. The method of claim 11, whereinreceiving the audio data comprises receiving telephone audio data, musicaudio data, or a combination thereof.
 13. The method of claim 11,wherein processing each audio processing task segment of the pluralityof audio processing task segments comprises performing audio datacompression, audio data decompression, audio data equalization, audiodata mixing, or a combination thereof.
 14. The method of claim 11,wherein assigning the respective priority level to each of the pluralityof audio processing task segments comprises assigning a priority levelaccording to a respective sampling rate of each of the plurality ofaudio processing task segments.
 15. The method of claim 14, whereinassigning the priority level according to the respective sampling ratecomprises determining a first sampling rate when the audio data segmentscomprise telephone audio data and determining a second sampling ratewhen the audio data segments comprise music audio data, wherein thefirst sampling rate is less than the second sampling rate.
 16. Themethod of claim 11, wherein assigning the respective priority level toeach of the plurality of audio processing task segments comprisesassigning a priority level to telephone audio data according to arespective processing time delay associated with the telephone audiodata.
 17. An electronic device, comprising, a digital signal processor(DSP), comprising: an input buffer configured to receive audio data; aprogrammable memory configured to interface with a core processor of theDSP to: partition the audio data into audio data segments, wherein theaudio data segments correspond to a plurality of audio processing tasksegments; and assign a respective priority level to each of theplurality of audio processing task segments; wherein the core processoris configured to process each audio processing task segment of theplurality of audio processing task segments according to the respectivepriority level of each of the plurality of audio processing tasksegments; and; an output buffer configured to store the audio datasegments.
 18. The electronic device of claim 17, wherein the DSP isconfigured to switch between the plurality of audio processing tasksegments based at least in part on whether the audio data segmentscomprise one of telephone audio data or music audio data.
 19. Theelectronic device of claim 17, wherein programmable memory is configuredto interface with the core processor to assign a higher priority levelto the telephone audio data than the priority level assigned to themusic audio data.
 20. The electronic device of claim 17, wherein thecore processor is configured to process each audio processing tasksegment of the plurality of audio processing task segments by processinga first segment of telephone audio data samples, a second segment ofmusic audio data samples, or a combination thereof.
 21. The method ofclaim 11, wherein partitioning the audio data into audio data segmentscomprises dividing each of the plurality of audio processing tasksegments into a series of smaller task segments to increase a speed atwhich each of the plurality of audio processing task segments isprocessed.